The MAHNOB Mimicry Database: A database of naturalistic human interactions

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. 1 Highlights • We present an audiovisual dataset for investigation of mimicry behaviour. • We report baseline performances from per-session mimicry classification experiments. • Performance is session-dependent, due to variability in subject expressiveness. • Current mimicry classification methods need more development for spontaneous data. ARTICLE INFO ABSTRACT Article history: Received xxxx Received in final form xxxx Accepted xxxx Available online xxxx Keywords: Behavioral mimicry Motor mimicry Temporal modelling Social signal processing People mimic verbal and nonverbal expressions and behavior of their counterparts in various social interactions. Research in psychology and social sciences has shown that mimicry has the power to influence social judgment and various social behaviours, including negotiation and debating, courtship, em-pathy and helping behaviour. Hence, automatic recognition of mimicry behaviour would be a valuable tool in various domains, and especially in negotiation skills enhancement and medical help provision training. In this work, we present the MAHNOB Mimicry database, a set of fully synchronised, mul-ti-sensory, audiovisual recordings of naturalistic dyadic interactions, suitable for investigation of mimicry and negotiation behaviour. The database contains 11 hours of recordings, split over 54 sessions of dyadic interactions between 12 confederates and their 48 counterparts, being engaged either in a socio-political discussion or negotiating a tenancy agreement. To provide a benchmark for efforts in machine understanding of mimicry behaviour, we report a number of baseline experiments based on visual data only. Specifically, we consider face and head movements, and report on binary classification of video sequences into mimicry and non-mimicry categories based on the following widely-used methodologies: two similarity-based methods (cross correlation and time warping), and a state-of-the-art temporal classifier (Long Short Term Memory Recurrent Neural Network). The best reported results are session-dependent , and affected by the sparsity of positive examples in the data. This suggests that there is much room for improvement upon the reported baseline experiments. Research in psychology has found that people mimic postures , facial expressions, mannerisms and other verbal and nonverbal expressions …